03/11/2024 - 09/11/2024

06/11/2024 01:47

I asked a question on the xilinx forums which basically boiled down to "use axi interconnect instead of axi smart connect". So I did that and created this diagram.

206bd08b697b7446d5af8ff2d0c92012.png

06/11/2024 01:52

After generating and exporting a bitstream of the above block design, I was able pass the ddr3 memory tests by programming the FPGA with vitis.

--Starting Memory Test Application--
NOTE: This application runs with D-Cache disabled.As a result, cacheline requests will not be generated
Testing memory region: mig_7series_0_memaddr
    Memory Controller: mig_7series_0
         Base Address: 0x80000000
                 Size: 0x80000000 bytes
          32-bit test: PASSED!
          16-bit test: PASSED!
           8-bit test: PASSED!
--Memory Test Application Complete--
Successfully ran Memory Test Application
--Starting Memory Test Application--
NOTE: This application runs with D-Cache disabled.As a result, cacheline requests will not be generated
Testing memory region: mig_7series_0_memaddr
    Memory Controller: mig_7series_0
         Base Address: 0x80000000
                 Size: 0x80000000 bytes
          32-bit test: PASSED!
          16-bit test: PASSED!
           8-bit test: PASSED!
--Memory Test Application Complete--
Successfully ran Memory Test Application

06/11/2024 02:21

I uploaded the bitstream to the device as a bin file and I cannot see the device under lspci like I normally can. I do see the device light up, so it's not a connection issue.

06/11/2024 02:34

I made a new block design by just deleting the second AXI interconnect and running connection automation

16a9a19acc3e01d021935d0e2d777593.png

It basically just uses one big axi interconnect instead of two. I'm not sure how it knows which AXI traffic to route where...

06/11/2024 02:36

The above diagram has the exact same issue.

06/11/2024 03:01

I discovered a grave issue with the above diagram. The port names are incorrect, this means the constraint file used is not accurate, this is what we used:

set_property -dict {PACKAGE_PIN K6} [get_ports pcie_refclk_clk_p]
set_property -dict {PACKAGE_PIN K5} [get_ports pcie_refclk_clk_n]
set_property -dict {PACKAGE_PIN R3} [get_ports {pcie_rxn[3]}]
set_property -dict {PACKAGE_PIN R4} [get_ports {pcie_rxp[3]}]
set_property -dict {PACKAGE_PIN N3} [get_ports {pcie_rxn[2]}]
set_property -dict {PACKAGE_PIN N4} [get_ports {pcie_rxp[2]}]
set_property -dict {PACKAGE_PIN L3} [get_ports {pcie_rxn[1]}]
set_property -dict {PACKAGE_PIN L4} [get_ports {pcie_rxp[1]}]
set_property -dict {PACKAGE_PIN J3} [get_ports {pcie_rxn[0]}]
set_property -dict {PACKAGE_PIN J4} [get_ports {pcie_rxp[0]}]

set_property -dict {PACKAGE_PIN P1} [get_ports {pcie_txn[3]}]
set_property -dict {PACKAGE_PIN P2} [get_ports {pcie_txp[3]}]
set_property -dict {PACKAGE_PIN M1} [get_ports {pcie_txn[2]}]
set_property -dict {PACKAGE_PIN M2} [get_ports {pcie_txp[2]}]
set_property -dict {PACKAGE_PIN K1} [get_ports {pcie_txn[1]}]
set_property -dict {PACKAGE_PIN K2} [get_ports {pcie_txp[1]}]
set_property -dict {PACKAGE_PIN H1} [get_ports {pcie_txn[0]}]
set_property -dict {PACKAGE_PIN H2} [get_ports {pcie_txp[0]}]

set_property -dict {PACKAGE_PIN E21 IOSTANDARD LVCMOS33} [get_ports pcie_reset]

set_property BITSTREAM.CONFIG.CONFIGRATE 16 [current_design]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]`
set_property -dict {PACKAGE_PIN K6} [get_ports pcie_refclk_clk_p]
set_property -dict {PACKAGE_PIN K5} [get_ports pcie_refclk_clk_n]
set_property -dict {PACKAGE_PIN R3} [get_ports {pcie_rxn[3]}]
set_property -dict {PACKAGE_PIN R4} [get_ports {pcie_rxp[3]}]
set_property -dict {PACKAGE_PIN N3} [get_ports {pcie_rxn[2]}]
set_property -dict {PACKAGE_PIN N4} [get_ports {pcie_rxp[2]}]
set_property -dict {PACKAGE_PIN L3} [get_ports {pcie_rxn[1]}]
set_property -dict {PACKAGE_PIN L4} [get_ports {pcie_rxp[1]}]
set_property -dict {PACKAGE_PIN J3} [get_ports {pcie_rxn[0]}]
set_property -dict {PACKAGE_PIN J4} [get_ports {pcie_rxp[0]}]

set_property -dict {PACKAGE_PIN P1} [get_ports {pcie_txn[3]}]
set_property -dict {PACKAGE_PIN P2} [get_ports {pcie_txp[3]}]
set_property -dict {PACKAGE_PIN M1} [get_ports {pcie_txn[2]}]
set_property -dict {PACKAGE_PIN M2} [get_ports {pcie_txp[2]}]
set_property -dict {PACKAGE_PIN K1} [get_ports {pcie_txn[1]}]
set_property -dict {PACKAGE_PIN K2} [get_ports {pcie_txp[1]}]
set_property -dict {PACKAGE_PIN H1} [get_ports {pcie_txn[0]}]
set_property -dict {PACKAGE_PIN H2} [get_ports {pcie_txp[0]}]

set_property -dict {PACKAGE_PIN E21 IOSTANDARD LVCMOS33} [get_ports pcie_reset]

set_property BITSTREAM.CONFIG.CONFIGRATE 16 [current_design]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]`

But we have no port pcie or pcie_reset!
So I changed the following port names:

pci_express_x4 --> pcie
pcie_perstn --> pcie_reset

I also went back to the first diagram, so the new diagram looks like:

b7b28e7fa4121f3c71c53d030f053502.png

06/11/2024 03:07

After those adjustments, I'm now able to see the device and pass memory tests!

lspci output:

[root@dhcp-10-163-102-46 scripts]# lspci -vv | grep -A 35 "Xilinx"
04:00.0 Serial controller: Xilinx Corporation Device 7024 (prog-if 01 [16450])
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 41
        Region 0: Memory at f5ff0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
[root@dhcp-10-163-102-46 scripts]#
[root@dhcp-10-163-102-46 scripts]# lspci -vv | grep -A 35 "Xilinx"
04:00.0 Serial controller: Xilinx Corporation Device 7024 (prog-if 01 [16450])
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 41
        Region 0: Memory at f5ff0000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
[root@dhcp-10-163-102-46 scripts]#

loading driver:

[root@dhcp-10-163-102-46 tests]# ./load_driver.sh
interrupt_selection .
xdma                  167936  0
Loading driver...insmod xdma.ko interrupt_mode=2 ...

The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@dhcp-10-163-102-46 tests]#
[root@dhcp-10-163-102-46 tests]# ./load_driver.sh
interrupt_selection .
xdma                  167936  0
Loading driver...insmod xdma.ko interrupt_mode=2 ...

The Kernel module installed correctly and the xmda devices were recognized.
DONE
[root@dhcp-10-163-102-46 tests]#

Running tests using my library:

[root@dhcp-10-163-102-46 scripts]# ./run_test.sh -a 0x80000000
Write operation successful. Time taken: 0.040745 seconds. Speed: 0.00299596 MB/s
Read operation successful. Time taken: 0.000196852 seconds. Speed: 0.620112 MB/s
Data verification succeeded
Hex dump of read data:
54 72 61 6e 73 66 65 72 20 74 65 73 74 3a 30 20
53 69 7a 65 3a 31 32 38 20 20 20 20 20 20 20 20
43 68 61 6e 6e 65 6c 3a 30 20 20 20 20 20 20 20
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
Number of h2c channels: 2
Number of c2h channels: 2
The PCIe DMA core is memory-mapped.
[root@dhcp-10-163-102-46 scripts]#
[root@dhcp-10-163-102-46 scripts]# ./run_test.sh -a 0x80000000
Write operation successful. Time taken: 0.040745 seconds. Speed: 0.00299596 MB/s
Read operation successful. Time taken: 0.000196852 seconds. Speed: 0.620112 MB/s
Data verification succeeded
Hex dump of read data:
54 72 61 6e 73 66 65 72 20 74 65 73 74 3a 30 20
53 69 7a 65 3a 31 32 38 20 20 20 20 20 20 20 20
43 68 61 6e 6e 65 6c 3a 30 20 20 20 20 20 20 20
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
31 36 42 20 45 78 61 6d 70 6c 65 20 4c 69 6e 65
Number of h2c channels: 2
Number of c2h channels: 2
The PCIe DMA core is memory-mapped.
[root@dhcp-10-163-102-46 scripts]#

07/11/2024 14:47

Triumf registration:
https://indico.psi.ch/event/16647/registrations/2402/?token=81826272-bc17-400f-96ba-672a17a045f3

Travel funding potential:
UK GSC - $400
Huffaker - $500

07/11/2024 14:54

How to correctly write a midas client, forum post and answer:

https://daq00.triumf.ca/elog-midas/Midas/2885